Accuracy Assessment of forecasting services

English: A service system is a dynamic configuration of people, technologies, organisations and shared information that create and deliver value to customers and other stakeholders [1]. The following cases are examples of customers receiving a service: taking a bus to go somewhere, or going to a restaurant to have a meal, or for a small IT (information technology) company, contracting a service to a bigger one in order to save costs and time. Service-oriented architecture (SOA) has become more popular during last years. Basically, this emerging development paradigm allows service providers to offer loosely coupled services. These services are normally only owned by the providers. As a result, the service user or client does not have to worry about the development, maintenance, infrastructure, or any other issue of how the service is working. To sum up, the user just has to find and choose the proper service. On the one hand, it presents several advantages. Firstly, common functionality can be contracted as a service in order to be able to focus on the own core missions. Secondly, it decreases the cost, since it is cheaper to contract a service than creating it yourself. Thirdly, clients take benefit of provider’s latest technologies. On the other hand, there is one big drawback: lack of trust. When you contract a service, you lose the direct control, the provider has access to your own data, you depend on him, and you experiment delays since your functionality is not working in-home. That is why the user has to decide previously which service is the most appropriate for his needs. Each client has different needs: quality (it varies among services), reputation (a famous or recommended provider usually gives more confidence), speed (agreements not to break thresholds), security (contract and trust in the provider), personalisation (preferential treatment from the provider), and locality (law is not the same in all countries). Therefore, a customer needs to know about the best service(s).Among all kind of services, we concentrate on forecasting services. Forecasting services show in advance a condition or occurrence about the future. There are plenty of domains: weather forecasts, stock market prices, results in betting shops, elections… Let us see a domain which is really familiar to all of us: weather forecast. When we are planning to travel, going somewhere or just deciding what to wear first thing in the morning, we wonder about weather conditions. To make these decisions, we check the weather forecast on TV news, a thermometer, or on a web site. However, sometimes we check several predictions and they do not agree. Which one will be the most accurate? Our goal in this master thesis is to assess the accuracy of these forecasting services in order to help prospective users to choose the best one according to their needs. To do it, we are going to compare forecast predictions with actual real observations

Mercury: using the QuPreSS reference model to evaluate predictive services

Nowadays, lots of service providers offer predictive services that show in advance a condition or occurrence about the future. As a consequence, it becomes necessary for service customers to select the predictive service that best satisfies their needs. The QuPreSS reference model provides a standard solution for the selection of predictive services based on the quality of their predictions. QuPreSS has been designed to be applicable in any predictive domain (e.g., weather forecasting, economics, and medicine). This paper presents Mercury, a tool based on the QuPreSS reference model and customized to the weather forecast domain. Mercury measures weather predictive services' quality, and automates the context-dependent selection of the most accurate predictive service to satisfy a customer query. To do so, candidate predictive services are monitored so that their predictions can be eventually compared to real observations obtained from a trusted source. Mercury is a proof-of-concept of QuPreSS that aims to show that the selection of predictive services can be driven by the quality of their predictions. Throughout the paper, we show how Mercury was built from the QuPreSS reference model and how it can be installed and used.Peer ReviewedPostprint (author's final draft

A Quality Model for Actionable Analytics in Rapid Software Development

Background: Accessing relevant data on the product, process, and usage perspectives of software as well as integrating and analyzing such data is crucial for getting reliable and timely actionable insights aimed at continuously managing software quality in Rapid Software Development (RSD). In this context, several software analytics tools have been developed in recent years. However, there is a lack of explainable software analytics that software practitioners trust. Aims: We aimed at creating a quality model (called Q-Rapids quality model) for actionable analytics in RSD, implementing it, and evaluating its understandability and relevance. Method: We performed workshops at four companies in order to determine relevant metrics as well as product and process factors. We also elicited how these metrics and factors are used and interpreted by practitioners when making decisions in RSD. We specified the Q-Rapids quality model by comparing and integrating the results of the four workshops. Then we implemented the Q-Rapids tool to support the usage of the Q-Rapids quality model as well as the gathering, integration, and analysis of the required data. Afterwards we installed the Q-Rapids tool in the four companies and performed semi-structured interviews with eight product owners to evaluate the understandability and relevance of the Q-Rapids quality model. Results: The participants of the evaluation perceived the metrics as well as the product and process factors of the Q-Rapids quality model as understandable. Also, they considered the Q-Rapids quality model relevant for identifying product and process deficiencies (e.g., blocking code situations). Conclusions: By means of heterogeneous data sources, the Q-Rapids quality model enables detecting problems that take more time to find manually and adds transparency among the perspectives of system, process, and usage.Comment: This is an Author's Accepted Manuscript of a paper to be published by IEEE in the 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2018. The final authenticated version will be available onlin

Energy Consumption of Automated Program Repair

Automated program repair (APR) aims to automatize the process of repairing software bugs in order to reduce the cost of maintaining software programs. Moreover, the success (given by the accuracy metric) of APR approaches has been increasing in recent years. However, no previous work has considered the energy impact of repairing bugs automatically using APR. The field of green software research aims to measure the energy consumption required to develop, maintain and use software products. This paper combines, for the first time, the APR and Green software research fields. We have as main goal to define the foundation for measuring the energy consumption of the APR activity. For that, we present a set of metrics specially crafted to measure the energy consumption of APR tools and a generic methodology to calculate them. We instantiate the methodology in the context of Java program repair. We measure the energy consumption of 10 program repair tools trying to repair real bugs from Defects4J, a set of real buggy programs. The initial results from this experiment show the existing trade-off between energy consumption and the ability to correctly repair bugs: Some APR tools are capable of achieving higher accuracy by spending less energy than other tools

Teaching MLOps in Higher Education through Project-Based Learning

Building and maintaining production-grade ML-enabled components is a complex endeavor that goes beyond the current approach of academic education, focused on the optimization of ML model performance in the lab. In this paper, we present a project-based learning approach to teaching MLOps, focused on the demonstration and experience with emerging practices and tools to automatize the construction of ML-enabled components. We examine the design of a course based on this approach, including laboratory sessions that cover the end-to-end ML component life cycle, from model building to production deployment. Moreover, we report on preliminary results from the first edition of the course. During the present year, an updated version of the same course is being delivered in two independent universities; the related learning outcomes will be evaluated to analyze the effectiveness of project-based learning for this specific subject.Comment: Accepted in 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET

Towards guidelines for building a business case and gathering evidence of software reference architectures in industry

Background: Software reference architectures are becoming widely adopted by organizations that need to support the design and maintenance of software applications of a shared domain. For organizations that plan to adopt this architecture-centric approach, it becomes fundamental to know the return on investment and to understand how software reference architectures are designed, maintained, and used. Unfortunately, there is little evidence-based support to help organizations with these challenges. Methods: We have conducted action research in an industry-academia collaboration between the GESSI research group and everis, a multinational IT consulting firm based in Spain. Results: The results from such collaboration are being packaged in order to create guidelines that could be used in similar contexts as the one of everis. The main result of this paper is the construction of empirically-grounded guidelines that support organizations to decide on the adoption of software reference architectures and to gather evidence to improve RA-related practices. Conclusions: The created guidelines could be used by other organizations outside of our industry-academia collaboration. With this goal in mind, we describe the guidelines in detail for their use.Peer ReviewedPostprint (published version

Which Design Decisions in AI-enabled Mobile Applications Contribute to Greener AI?

Background: The construction, evolution and usage of complex artificial intelligence (AI) models demand expensive computational resources. While currently available high-performance computing environments support well this complexity, the deployment of AI models in mobile devices, which is an increasing trend, is challenging. Mobile applications consist of environments with low computational resources and hence imply limitations in the design decisions during the AI-enabled software engineering lifecycle that balance the trade-off between the accuracy and the complexity of the mobile applications. Objective: Our objective is to systematically assess the trade-off between accuracy and complexity when deploying complex AI models (e.g. neural networks) to mobile devices, which have an implicit resource limitation. We aim to cover (i) the impact of the design decisions on the achievement of high-accuracy and low resource-consumption implementations; and (ii) the validation of profiling tools for systematically promoting greener AI. Method: This confirmatory registered report consists of a plan to conduct an empirical study to quantify the implications of the design decisions on AI-enabled applications performance and to report experiences of the end-to-end AI-enabled software engineering lifecycle. Concretely, we will implement both image-based and language-based neural networks in mobile applications to solve multiple image classification and text classification problems on different benchmark datasets. Overall, we plan to model the accuracy and complexity of AI-enabled applications in operation with respect to their design decisions and will provide tools for allowing practitioners to gain consciousness of the quantitative relationship between the design decisions and the green characteristics of study.Comment: Accepted as registered report at ESEM 202

Accuracy Assessment of forecasting services

English: A service system is a dynamic configuration of people, technologies, organisations and shared information that create and deliver value to customers and other stakeholders [1]. The following cases are examples of customers receiving a service: taking a bus to go somewhere, or going to a restaurant to have a meal, or for a small IT (information technology) company, contracting a service to a bigger one in order to save costs and time. Service-oriented architecture (SOA) has become more popular during last years. Basically, this emerging development paradigm allows service providers to offer loosely coupled services. These services are normally only owned by the providers. As a result, the service user or client does not have to worry about the development, maintenance, infrastructure, or any other issue of how the service is working. To sum up, the user just has to find and choose the proper service. On the one hand, it presents several advantages. Firstly, common functionality can be contracted as a service in order to be able to focus on the own core missions. Secondly, it decreases the cost, since it is cheaper to contract a service than creating it yourself. Thirdly, clients take benefit of provider’s latest technologies. On the other hand, there is one big drawback: lack of trust. When you contract a service, you lose the direct control, the provider has access to your own data, you depend on him, and you experiment delays since your functionality is not working in-home. That is why the user has to decide previously which service is the most appropriate for his needs. Each client has different needs: quality (it varies among services), reputation (a famous or recommended provider usually gives more confidence), speed (agreements not to break thresholds), security (contract and trust in the provider), personalisation (preferential treatment from the provider), and locality (law is not the same in all countries). Therefore, a customer needs to know about the best service(s).Among all kind of services, we concentrate on forecasting services. Forecasting services show in advance a condition or occurrence about the future. There are plenty of domains: weather forecasts, stock market prices, results in betting shops, elections… Let us see a domain which is really familiar to all of us: weather forecast. When we are planning to travel, going somewhere or just deciding what to wear first thing in the morning, we wonder about weather conditions. To make these decisions, we check the weather forecast on TV news, a thermometer, or on a web site. However, sometimes we check several predictions and they do not agree. Which one will be the most accurate? Our goal in this master thesis is to assess the accuracy of these forecasting services in order to help prospective users to choose the best one according to their needs. To do it, we are going to compare forecast predictions with actual real observations

Training future ML engineers: a project-based course on MLOps

Recently, the proliferation of commercial ML-based services has given rise to new job roles, such as ML engineers. Despite being highly sought-after in the job market, ML engineers are difficult to recruit, possibly due to the lack of specialized academic curricula for this position at universities. To address this gap, in the past two years, we have supplemented traditional Computer Science and Data Science university courses with a project-based course on MLOps focused on the fundamental skills required of ML engineers. In this paper, we present an overview of the course by showcasing a couple of sample projects developed by our students. Additionally, we share the lessons learned from offering the course at two different institutions.This work is partially supported by the NRRP Initiative – Next Generation EU ("FAIR - Future Artificial Intelligence Research", code PE00000013, CUP H97G22000210007); the Complementary National Plan PNC-I.1 ("DARE - DigitAl lifelong pRevEntion initiative", code PNC0000002, CUP B53C22006420001), and the project TED2021- 130923B-I00, funded by MCIN/AEI/10.13039/50110001 1033 and the European Union Next Generation EU/PRTR.Peer ReviewedPostprint (author's final draft